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ABSTRACT 

The rapid growth of using Web 2.0 tools such as blogs has increased online courses in 
education. Questioners are the most commonly used instruments to assess students' 
attitudes toward the online courses. This study provides a set of specific guidelines that 
the researchers used to develop a questionnaire to measure students' attitudes toward 
the course blog. It focuses on test construction and instrument validation as two 
primary stages in developing the questionnaire and details a series of steps nested 
within two stages. Participants were 30 undergraduate students enrolled in a course 
blog. To analyze the data, qualitative findings of interview were complemented by 
statistical results from quantitative data. To improve content adequacy and internal 
validity of the instrument items, 25 students who took part in piloting the instrument 
were interviewed. We carried out Statistical analysis to evaluate inter-item correlations, 
and reliability alpha coefficient of the instrument items. The guidelines applied in this 
study can be used in other studies to develop a valid instrument to measure other 
constructs, particularly when a researcher does not have access to a large sample. 
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INTRODUCTION 

The rapid growth of using Web 2.0 tools such as blogs has increased online courses in 
education. Research shows that students' attitude toward online courses is a significant 
factor which may affect their learning (Simsek, 2008). Positive attitudes enhance 
learners' motivation to learn and retain information in particular circumstances, while 
negative attitudes may result in resisting learning (Duda & Garrett, 2008). 

Giving a definition to attitude has always been a perennial problem because of its 
construct which implies a learner's way of thinking positively or negatively (Lopper, 
2006). To assess students' attitudes in a learning environment, questionnaires are 
considered as a reliable instrument (DeVellis 2003; Colosi, 2006; Radhakrishna, 2007). 
There are three common approaches in applying questionnaires. The first is to select and 
apply a questionnaire that has been previously developed and used by other researchers. 
The second approach is to develop a questionnaire through the modification of an 
existing questionnaire. 

The third is to develop a new questionnaire due to a lack of suitable questionnaire to 
explore the constructs that have not been investigated in previous research (Estabrooks 
& Wallin, 2004). 
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The third approach deems more reliable than others because developing questionnaire 
reduces measurement error; however, inadequate attention to psychometric assessment 
and reliability evaluation is considered major obstacles for instrument developers in this 
approach (Estabrooks & Wallin, 2004). For example, to assess students' attitudes toward 
blogs, researchers used a self developed instrument without reporting anything on 
reliability and validity as vital entities to develop a reliable and valid instrument 
(Pinkman, 2005). This study attempts to provide a set of particular guidelines to develop 
a questionnaire. It also exemplifies the application of these guidelines in assessing 
students' attitudes toward the course blog. The main reason for conducting the course 
blog was the frequent use of blogs as a social essence of Web 2.0 tools in education 
(Kurt, Izmirli, & Sahin-Izmirli, 2011). Blogs allow students to act autonomously, to 
improve their motivation, productivity, cultural knowledge, language, and 
communication (Rezaee & Oladi, 2008). Students can interact with other students and 
their teacher anywhere and anytime (Tu, Chen, & Lee, 2007). They can also edit or omit 
whatever they have posted on blogs (Johnson, 2004). However, understanding blog 
facilities alone cannot fully explain students' effective application of the course blog, 
unless we consider students' attitudes in a learning environment (Shahsavar & Tan, 
2011 ). 

Guidelines in instrument development 

Particular guidelines are required to develop a valid instrument (Burton & Mazerolle, 
2011). This study provides a set of specific guidelines focused on two stages and a series 
of steps nested within them to develop an instrument (see Figure:!). 



Figure 1: 

A flow diagram on an instrument development 

These guidelines are set out by the researchers after a comprehensive review of 
available guidelines in scale development (e.g., Delamere, Wankel, & Hinch, 2001; 
DeVellis, 2003; Ping, 2005; Burton & Mazerolle, 2011; name as a few). 
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Stage One: Test Construction 

As Figure 1 shows, stage 1 describes the process of moving from conceptualization levels 
to item constructions in developing the instrument. This stag seeks to gather input on 
test construction in three steps: defining construct, item generation, determining the 
format (see Figure 1). 

Defining the construct is the most significant step for developing items. Instrument 
developers require deep thinking about the construct. To this end, they have to explore 
the relevant literature and also focus on the research domain. In some cases, an 
instrument may exist from a particular domain which may not be applicable to other 
domains (Burton & Mazerolle, 2011). For example, an instrument may exist to measure 
students' attitudes toward a blog as a learning tool which may not be applicable to 
assess students' attitudes toward the course blog. 

Generating an item pool represents the instrument concept and captures its construct 
essence (Delamere, 1997; DeVellis, 2003; Ping, 2005). In this step, instrument 
developers should generate items and examine them for different item characteristics 
such as, ambiguous pronouns, misplaced modifiers (DeVellis, 2003), avoidance of leading 
items, abstract terms and jargon, vague statement, multiple negative, and double 
barreled (Office of Educational Assessment, 2006). 


Determining the format of the instrument is the last step that occurs with item 
generation simultaneously because the two are compatible (DeVellis, 2003). Various 
formats exist to present items in the instrument: Thunderstone scaling, Guttman scaling, 
Likert scale, semantic differential, visual analog scale, binary options, time frames, and 
unipolar versus bipolar scale (see DeVellis, 2003; Office of Educational Assessment, 
2006, for more information on each format). Instrument developers should select the 
format with respect to the instrument construct. For example, since a Likert scale 
specifies the level of agreement or disagreement of respondents, it is regarded as the 
most reliable format extensively applied to measure attitude (Zan & Martino, 2007). 

Stage Two: Instrument Validation 

Stage 2 presents the instrument validation in three steps: item judgment, pilot test the 
instrument, and instrument assessment (see Figure: 1). 

Item judgment serves different purposes to maximize the instrument face validity and 
content-related validity as the primary steps in establishing construct validity (Burton & 
Mazerolle, 2011). 

In this ste, to ensure instrument developers that items are clearly worded, a panel of 
experts reviews an item pool to evaluate the clarity, readability and content validity of 
items (Delamere, et al., 2001; Mayfield & Crompton, 1995). 

The "Delphi Technique" is one of the most common methods to gain experts' knowledge, 
ideas, and agreement about the item pool (Delamere et al., 2001, p. 13). In this method, 
each panel expert reviews item pool independently in two or more rounds to evaluate 
and give comments on clarity, readability and content validity of items. After each round, 
the instrument developer provides an anonymous summary of experts' feedback and 
reasons they provided for their judgments to develop revised item pool. However, the 
instrument developer is responsible to accept or reject experts' advice to retain, remove, 
synthesize or change items in each round. The summary and revised items will be 
returned to experts for another round. 
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The panel applies the same procedure on item pool and the instrument developer makes 
amendments accordingly. The process will be stopped after receiving a panel 
confirmation on a final feedback report and items (Delamere et al., 2001). 

The second step of instrument validation is to pilot test an instrument. An appropriate 
sample size and subjects should be used for pilot testing an instrument. In addition, 
instrument developers should select the sample from the population of interest for the 
study. Participants' demographic profile s such as their level of experience or education 
should well match the subjects' profile of the main study (Burton & Mazerolle, 2011). 

The last and most fundamental step of instrument validation is instrument assessment. 
Instrument developers can evaluate and analyze the performance of individuals on each 
item by conducting the interview, using item scale correlation to assess correlation 
among items (Field, 2000), coefficient alpha to measure the internal consistency or 
reliability for scale items (Oppenheim; 1992). However, factor analysis has also been 
used in many studies in order to purify the scale items and reduce the number of items 
without scarifying instrument reliability (Burton & Mazerolle, 2011). 

This study attempts to exemplify how the aforementioned guidelines are used to develop 
a questionnaire to measure students' attitudes toward the course blog. 

METHOD 

Participants 

In this study, 30 undergraduate students took part in the course blog to pilot test a 
questionnaire. Students aged between 20 and 25 enrolled in an obligatory course in the 
first semester of 2010. All spoke English as a second language. Most students had 
personal computers and home or dormitory Internet access. 

Procedure 

In this study, to develop the questionnaire measuring students' attitudes toward the 
course blog, the researchers applied a set of particular guidelines presented in Figure 1. 
The finalized course blog attitude questionnaire (CBAQ) is presented in Appendix. 

First, we constructed the test by describing lines of thought addressed us to define 
attitude such as: What is attitude? What does students' attitude to the course blog 
mean? To arrive at precise answers for these questionings, we carried out a 
comprehensive review of Web 2.0 tool attitude instruments: Loyd and Gressard (1984), 
Shih and Gamon (2001), Cheong and Cheung (2008), Duda and Garrett (2008), 
Shahsavar, Tan, and Aryadoust (2010), and 31 computer attitude instruments presented 
by Shaft, Sharfman, and Wu (2004). 

After articulating the purpose of the instrument, we generated a pool of 30 items. We 
examined items based on item characteristics as pointed out earlier in item generation. 
Moreover, to boost the internal consistency, we used some redundant items (see item 5 
& item 19 in Appendix). We also advocated using four-point Likert-type scale ranging 
between 1 (e.g., strongly disagrees) and 4 (e.g., strongly agree). 

After generating a pool of appropriate items and selecting a format for instrument, we 
applied item judgment to maximize the instrument validation. Delphi Technique was 
applied to analyze items critically by a panel of experts. The panel was composed of six 
individuals who had worked with developing the questionnaire in an online learning 
environment and also been familiar with blogs. 
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Each panel expert reviewed the item pool independently in two rounds to evaluate and 
give comments on clarity, readability and content validity of items. At the end of the first 
round, we provided an anonymous summary of experts' feedback and reasons they 
provided for their judgments (see Table: 1). 


Table: 1 

Examples of item judgment by expert panels in two rounds 


# 

Item pool 

Experts' comment 

1 

Blogs make the class more interesting 
to me. 

-This item seems to evaluate interest. 

(Rl) 

-Use "a course blog" instead of "blogs" 

(Rl) 

-Rephrasing the item. (R2) 

-Define the course blog? (Rl) 

2 

I have to do a difficult training course 
blog to understand how to use blogs. 

-Clarify training course on blogs. (R2) 

3 

Blogs help me to learn new terms and 
new ideas from my classmates. 

-Use a simpler word like "word" instead 
of "term". (R2) 

5 

I enjoy sharing my knowledge with 
my classmates on blogs. 

-Omit item 5 or item 19 as they are 
redundant. 

6 

I am not familiar with blogging*; 
therefore I did not find it interesting. 
(*blogging means writing on a blog) 

-The item is double-barreled. (Rl) 

7 

Using blogs is a waste of time for 
learning. 

-Reword the item. (Rl) 

8 

A course blog provides me with 
learning opportunities that I have 
never tried before in traditional 
classrooms. 

-What does traditional classroom mean? 
(Rl) 

9 

I feel isolated as a student when I 
take a course blog. 

-Rephrase the item. (R2) 

14 

Blogs help us to communicate and 
discuss more with other students. 

-The item is double-barreled. (Rl) 

15 

I feel aggressive and hostile toward 
using blogs in class. 

-The item is double-barreled. (Rl) 

-Hostile seems inappropriate in this 
context. (R2) 

16 

Blogs are good as a learning tool for 
thinking, arguing and discussing with 
others. 

-The item is three-barreled item. (Rl) 

19 

It is interesting to share our personal 
idea with others on blogs. 

- This item and item 5 are redundant, 
omit one. 

-"Our" is an ambiguous pronoun. (R2) 


Note. # = item numbers in the CBAQ presented in Appendix. Rl= experts' comment to item pool in the first 
round. R2 = experts' comment on item pool in the second round. 


The summary and revised items were returned to the experts for the second round. 
However, in both rounds, the researchers made a final decision on retaining, removing, 
or modifying items (DeVellis, 2003). For example, three experts referred to item 
redundancy between item 5 and 19 (see Table 1) and commented on omitting one of 
them but the researchers decided to keep them to increase the internal consistency. The 
process completed after receiving a panel confirmation on the second feedback report.At 
this stage, we kept 22 items that had been reviewed by experts and modified by the 
researchers accordingly. ->a 4 



This process allowed us to advance to instrument validation. To pilot the questionnaire, 
three face-to-face training sessions were conducted in a computer lab before students 
started practicing through a blog independently. During the first training session, 
students were briefed on the purpose of the study and they were told that their 
participation on blog was compulsory. After that they registered on the blog created by 
one of the researchers at http://www.bloqqer.com . While their blog display names were 
changed to protect their identity. Since few students did not have any experience in 
blogging, first they had some hands-on practice on basic blogging skills such as posting, 
and leaving comments on blog. This pilot study assisted the teacher in piloting the 
questionnaire to measure students' attitude in the main study. Twenty two items that 
had been reviewed by experts and modified were administered to students to precede 
the instrument validation. 

Data analysis and results 

To assess students' performance on items, qualitative findings of interview were 
complemented by statistical results from quantitative data.To improve content adequacy 
and internal validity of the instrument items, 25 students who took part in piloting the 
instrument were interviewed. Two kinds of interviews were conducted: a focus group 
interview and individual interview which took 20 and 10 minutes respectively. The 
interview questions were fixed and open-ended such as: How did you find the instrument 
items? Did you get the meaning of all items? Do you have any comments to improve the 
instrument?—Students' responses to interview questions were audio taped and then 
analyzed along with the researchers. Some students identified ambiguous and difficult 
items and expected for rewording, rephrasing, or removing them (see Table: 2). 

Table 2: 

Examples of Students' Reasons Comments on the Items 


# 

T 

2 

7 

8 


1 

5 

1 

7 


Item 


Students' comment 


Blogs make the course more interesting to me. 

I need to do a lot of training on blogs to understand how to use 
them. 

Using blogs for learning is a waste of time. 

A course blog provides me with learning opportunities that I 
have never tried before in traditional classrooms*. 

(*A traditional classroom refers to a teacher-fronted classroom 
that requires the physical presence of students in the same 
space and time.) 

I feel aggressive toward using blogs in class. 

It is easier to voice out my opinions through blogs than speak it 
face to face with my instructor. 


Rephrase the item. 

Rephrase the item. 

Reword the item. 

Remove "Traditional 

classrooms" definition. 


"Aggressive" is an ambiguous 
word. 

The item looks ambiguous. 


Note. # = item numbers presented in the CBAQ in Appendix. 


For example, most of them mentioned that they were familiar with traditional classroom 
definition and expected for omission the definition (see Table 2, item 8) or 20 
interviewees had problems in the meaning of "aggressive" and expected for rewording it 
(see Table 2, item 15). 


We carried out Statistical analysis to evaluate inter-item correlations, and reliability 
alpha coefficient of items by applying SPSS computer program. Version 16 (SPSS Inc., 
2007). Firstly, reverse coding was run to consider those items of the instrument whose 
correlations with other items were negative. Likewise, since some negative correlation 
among items may not be applicable by reverse coding (DeVellis, 2003), item scale 
correlations and the internal consistency were commuted and examined for items. 
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The inter-item correlation coefficients of items fell between 0.30-0.60 which was 
accepted as the moderate inter-item correlation (Burton & Mazerolle, 2011) and the 
internal consistency of items was 0.88 which shows good level of consistency (Cronbach 
& Richard, 2004). Considering all these factors, the CBAQ was developed to assess 
students' attitude to the course blog (see Appendix). 

CONCLUSION 


The rapid development of Web 2.0 tools has changed education by facilitating students' 
learning. Understanding these facilities alone cannot fully explain users' effective 
application of online courses unless the instructors consider students' attitudes to online 
courses (Simsek, 2008). This paper provides a set of guidelines to develop the 
instrument to measure students' attitudes to the course blog. The guidelines may be 
applied in other studies to develop an instrument to measure other constructs 
particularly when a researcher does not have access to a large sample. 

Limitations and further research 

Research shows that using reliability alpha coefficient and factor analysis in tandem 
would improve an instrument psychometric features and validity (Burton & Mazerolle, 
2011). Although factor analysis assists researchers in assessing the dimensionality of 
remaining items and purifying the instrument, it demands a large data set (Field, 2000). 
Some commentators refer to a ratio bigger than 5 respondents for each item as the 
minimum requirement of factor analysis (Hair, Black, Babin, & Anderson, 2010), while 
others recommended a minimum overall sample of 300 respondents, or a minimum of 10 
respondents for any instrument item (Burton & Mazerolle, 2011). 

In this study, even if the researchers had easy access to undergraduate students, some 
practical problems limited the desirability of a large subject: 

> the computer lab had approximately 35-40 seats for students to 
participate in the course blog activities in face-to-face sessions; 

> creating blog at www.bloqqer.com had maximum size limitations of 100 
users. Future studies, which apply other Web 2.0 tools in online courses 
by overcoming these limitations, will need to be undertaken. 
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APPENDIX 

The Course Blog Attitude Questionnaire 


# 

Statement 

S 

D 

A 

S 



D 



A 


1 A course blog* is very interesting to me. 

(*A course blog means a blog that has been set up by the 


teacher to facilitate students' online learning.) 

2 I need do a lot of training in a course blog to understand 
how to use blogs. 

3 Blogs help me learn new words from my classmates' 
postings. 

4 Blogs help me get new ideas from my classmates' 
postings. 

5 I enjoy sharing my knowledge with my classmates on 
blogs. 

6 As a new way of writing, I think blogging* is interesting. 
(*blogging means writing on a blog) 

7 Using blogs in a learning process is a waste of time. 

8 A course blog provides me with learning opportunities 
that I have never tried before in traditional classrooms. 

9 I felt isolated when I participated in a course blog. 

10 It is easier for me to convey my thoughts through blogs 
than to converse face-to-face with my classmates. 

11 A course blog is quite boring. 

12 I'm not the type to do well with blogs. 

13 I prefer working alone to working with other students in a 
course blog. 

14 Blogs help me to communicate more with other students. 

15 I do not like to use blogs. 

16 Blogs are good as a tool for discussing with other 
students. 

17 It is easier to voice out my opinions through blogs than to 
convey it face- to-face to my instructor. 

18 I'm motivated to read others' postings before blogging. 

19 It is interesting to share my personal ideas with others on 
blogs. 
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